Managing the Health of Security Experiments

Testbed experiments are a challenge to manage manually, because they involve multiple machines and their correctness depends on the correct operation of testbed infrastructure that is often hidden from the experimenter. Testbed experiments that recreate security events add management challenges of scale – they are often very large; complexity – many threats work only if certain conditions are met by the network environment; and risk – they often involve malicious code and disruptive actions that must be contained. Finally, an experiment may be run by someone who did not create it originally. It is challenging for this new experimenter to ascertain if any experiment behavior was intended or a sign of failure, and to diagnose and correct failures. We introduce a new paradigm of experiment health that denotes a user-supplied description of correct experiment behavior, i.e., healthy experiments behave as their creators intended. We then propose an experiment health management infrastructure that can be added to existing testbeds to improve their usability and robustness. The infrastructure consists of an expectation language in which a user expresses her notion of experiment health, a monitoring infrastructure that is driven by user expectations, health evaluators, recovery engines and a shared library of health tools and collected experiment statistics. This infrastructure is useful not only for experiment management, but also for testbed management.